Skip to content

feat(vm): support corporate HTTP forward proxy egress for microVM sandboxes - #3090

Merged
johntmyers merged 3 commits into
NVIDIA:mainfrom
feloy:proxy-vm
Sep 2, 2026
Merged

feat(vm): support corporate HTTP forward proxy egress for microVM sandboxes#3090
johntmyers merged 3 commits into
NVIDIA:mainfrom
feloy:proxy-vm

Conversation

@feloy

@feloy feloy commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Summary

Gives the MicroVM compute driver the same operator-owned corporate forward-proxy surface the Podman and Kubernetes drivers already have, so VM sandboxes on proxy-only corporate networks can reach the destinations policy allows. The supervisor-side machinery from #1792 is reused unchanged; what this adds is the per-driver half — configuration, validation, a driver→guest argument channel, and credential/CA delivery into the guest.

Related Issue

Closes #3088

Note for reviewers on issue state: #3088 currently carries only state:triage-needed — it has no state:accepted, no roadmap placement, and no agent:plan-requested/agent:implementation-requested label. This work was done on an explicit maintainer request rather than through the unattended queue. Per AGENTS.md that request authorizes the phase, and no lifecycle labels were changed. Flagging it because PRs for features are otherwise expected to link an accepted issue.

Changes

The argv channel (the one genuinely new mechanism). Podman and Kubernetes build the supervisor's command line in Rust. The VM guest init script runs as PID 1 and execs a fixed argv, and libkrun's krun_set_exec receives an empty argument array, so there was no channel for driver-owned supervisor arguments. The supervisor's proxy flags deliberately have no env = fallback, and build_guest_environment merges user-supplied environment, so the guest environment is not a safe transport either.

  • The driver writes /opt/openshell/supervisor-args into the overlay upperdir on every launch; the guest init script reads it verbatim, one argument per line (no word splitting, globbing, or expansion), and appends it to every supervisor exec.
  • It is written even when empty. That is what makes the channel unforgeable: the upperdir always shadows the read-only image layer, so a sandbox image can neither supply its own supervisor arguments by baking a file at that path nor disable the operator's by omitting one. This mirrors the existing init.d manifest, which solves the same trust problem for guest init drop-ins.
  • Because both launch backends exec the same init script, this lands once and covers libkrun and QEMU. runtime.rs is untouched.

Configuration. [openshell.drivers.vm] accepts https_proxy, no_proxy, proxy_auth_file, proxy_auth_allow_insecure, proxy_connect_by_hostname, and proxy_ca_bundle, with matching --https-proxy--proxy-ca-bundle flags and OPENSHELL_VM_* environment variables on the driver binary (needed independently, since the driver also runs as an external gRPC driver). The booleans travel as explicit values rather than presence flags so an operator's explicit false survives the gateway→driver hop and still trips the pairing checks. The keys are not inheritable from [openshell.gateway] and are rejected by template.driver_config.vm.

Shared validation, fail-closed. Added openshell_core::driver_utils::validate_upstream_proxy_settings holding the pairing rules the Podman driver established, called by both the gateway and the driver — without the gateway-side check an invalid table surfaces as an opaque driver-readiness timeout instead of an error naming the key. Podman and Kubernetes are deliberately left on their own copies to keep this PR scoped.

Credential and CA delivery. A microVM has no bind mounts or container secrets, so both are staged into the per-sandbox overlay the way the gateway JWT already is — credential root-only at 0600, CA at 0644, both at fixed /opt/openshell paths, both rewritten every launch (so removing a setting clears material a previous launch staged into a preserved overlay), and both removed with the sandbox state directory.

Docs. docs/reference/gateway-config.mdx (MicroVM section), docs/reference/sandbox-compute-drivers.mdx, crates/openshell-driver-vm/README.md, architecture/sandbox.md, and a VM corporate-proxy troubleshooting section in the debug-openshell-cluster skill.

Security considerations for review

  • Credential at rest in the overlay image. feat(vm): support corporate HTTP forward proxy egress for microVM sandboxes #3088 explicitly deferred this to review. The credential ends up inside the per-sandbox overlay.ext4 on the gateway host, which differs from the Podman secret model. It is taken because the per-sandbox gateway JWT — a bearer token for the gateway — already travels this exact path, so the exposure is not new; it is documented in architecture/sandbox.md and the config reference rather than left implicit. Alternatives considered and rejected as disproportionate: copying to tmpfs via an init drop-in and deleting from the overlay, or a new driver→guest vsock secret channel.
  • Reachability. Guest egress leaves through gvproxy, so a proxy on the gateway host's loopback is reachable only via host.openshell.internal (gvproxy NATs 192.168.127.254 to the host's 127.0.0.1). Documented, and exercised by the e2e. The guest→gateway callback is unaffected and never traverses the proxy.
  • Runtime capabilities unchanged. The VM driver still advertises no policy DNS or transparent TCP interception; this PR does not change what traffic reaches the supervisor's proxy.

Testing

  • mise run pre-commit passes
  • Unit tests added/updated
  • E2E tests added/updated (if applicable)

Unit — 29 new tests:

  • openshell-core: the shared validator (unsupported scheme, inline credentials, present-but-empty values, each auxiliary key without a proxy URL, the cleartext acknowledgement including the https:// exemption, CA bundle valid with a plain-HTTP intercepting proxy) and the three new guest paths in the control-root test.
  • openshell-driver-vm: argv builder emits guest paths and never the operator's host paths; Some(false) is not passed as a presence flag; the argument file renders empty when unconfigured and one-argument-per-line otherwise, preserving values containing spaces; newline/NUL values rejected; credential and CA validated with the supervisor's own validators and errors never echo file contents; build_guest_environment emits no proxy setting and no user-supplied environment can forge an --upstream-* argument; template.driver_config.vm rejects all six keys; Debug redacts the proxy URL and credential path.
  • openshell-server: spawn forwards the settings as driver argv and omits unset ones; invalid config is rejected before the driver starts, naming the key.

E2E — new e2e/rust/tests/vm_corporate_proxy.rs in the e2e:vm lane, the VM counterpart of podman_corporate_proxy.rs. Fixtures run as host processes (no container-runtime dependency for this lane) and are reached from the guest through host.openshell.internal, which is also what proves the documented host-loopback reachability rule. Three cases:

  1. A policy-approved HTTPS destination reached through the proxy — the proxy log records CONNECT 192.168.127.254:<port> auth=ok, proving both the validated-IP CONNECT target and that the overlay-staged credential reached the supervisor (the proxy answers 407 without it). The policy-denied destination never appears in the proxy log at all, and a port-qualified no_proxy destination is reached while never touching the proxy.
  2. An https:// proxy trusted via proxy_ca_bundle, with the CA minted by the fixture and recovered from its output.
  3. Fail-closed: no_proxy without https_proxy stops the gateway from serving, with the offending keys named in its log.

Verification performed on this branch:

Check Result
mise run pre-commit clean
cargo clippy --all-targets -- -D warnings (core, driver-vm, server, e2e) clean
mise run test:rust (full workspace incl. openshell-server) 0 failures
mise run e2e:vm exit 0 — smoke 1, host_gateway_alias 4, vm_gateway_start 1, vm_corporate_proxy 3

smoke passing is a deliberate regression signal: every sandbox now runs the argument reader before the supervisor exec, so a mistake in the reader or in the empty-array expansion under set -u would break ordinary VM boot, not just the proxy path.

Two notes for anyone running the VM lane locally on macOS, both discovered here and neither a product bug:

  • brew install e2fsprogs is required — the driver shells out to mke2fs on the host to build the guest rootfs. It is keg-only; the driver probes /opt/homebrew/opt/e2fsprogs/sbin directly.
  • VM sandbox main-process commands must stay short. libkrun passes the guest environment on the kernel command line and the driver base64-encodes the main-process spec into it, so a ~1000+ character command aborts the VM before boot with a libkrun TooLarge panic. E2E workloads of any size should use SandboxGuard::create (delivered via ExecSandbox) rather than create_keep_with_args.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

@copy-pr-bot

copy-pr-bot Bot commented Sep 1, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@johntmyers

Copy link
Copy Markdown
Collaborator

gator-agent

Blocked

Gator is blocked because PR #3090 currently has merge conflicts with main.

Next action: @feloy, please update the branch from main, resolve the conflicts, and push the resulting commit so code review can begin.

Gator metadata
  • Head SHA: 9a9acd95f207b930783839ca63133724870282dd
  • Base SHA: bb70461878ca5d152616961609db9b403b289098
  • Merge base SHA: 8a13bc1298e9f74f110f0bc57cfb12e91c93d9c7
  • Patch ID: 7e491a65332a9bb1710f117889c43992addd6c8a
  • Gator payload: 8
  • Next state: gator:blocked
  • Blocked reason: merge_conflict

@johntmyers johntmyers added the gator:blocked Gator is blocked by process or repository gates label Sep 2, 2026

@johntmyers johntmyers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

PR Review Status

The merge-conflict blocker from the previous head is resolved. The initial code review found two concrete blockers in the new VM proxy path: CA bundle staging can consume an unbounded special file and accepts certificates the guest will reject, and the published loopback recipe is incorrect for the supported QEMU/TAP backend.

Action required: @feloy, please address GATOR-dc1218b5-01 and GATOR-dc1218b5-02, then push an update.

Blocking findings:

  • GATOR-dc1218b5-01: bound and fully validate the host CA bundle before staging it.
  • GATOR-dc1218b5-02: make the host-loopback guidance accurate for both libkrun/gvproxy and QEMU/TAP.

Carried findings:

  • None
Gator metadata
  • Validation: Project-valid implementation of linked issue #3088 with a clear VM corporate-proxy user path, threat-boundary analysis, docs, and E2E coverage.
  • Docs: Updated, but the host-loopback contract is materially incorrect for QEMU/TAP.
  • Checks: Current-head Branch Checks and Helm Lint gates are pending; required branch workflows have not been dispatched.
  • E2E: test:e2e is required for VM networking and credential flow; dispatch is deferred until review blockers are resolved.
  • Head SHA: dc1218b51ac14f1315c1a633cae316ff45963549
  • Base SHA: 7b64c5c88ece0878592160cb86da5e0fd1f64158
  • Merge base SHA: 7b64c5c88ece0878592160cb86da5e0fd1f64158
  • Patch ID: e940265736870bac2a568c5f4c0be0ba145c2786
  • Gator payload: 8
  • Review mode: initial
  • Previous reviewed SHA: none
  • Review budget exhausted: no
  • Maintainer decision required: no
  • Next state: gator:in-review

Comment thread crates/openshell-driver-vm/src/driver.rs Outdated
Comment thread docs/reference/gateway-config.mdx Outdated
@johntmyers johntmyers added gator:in-review Gator is reviewing or awaiting PR review feedback and removed gator:blocked Gator is blocked by process or repository gates labels Sep 2, 2026
…dboxes

The corporate forward proxy machinery from NVIDIA#1792 is driver-agnostic and
already merged: openshell-supervisor-network implements CONNECT chaining,
NO_PROXY matching, credentials, https:// proxies and corporate CA trust, and
openshell-sandbox exposes it as six argv-only flags. Podman gained the driver
half in NVIDIA#2245/NVIDIA#2512 and Kubernetes in NVIDIA#2633; the VM driver had none of it, so
VM sandboxes on proxy-only networks could not reach any destination requiring
the proxy even when policy allowed it.

The blocking piece was not proxy logic but delivery: the VM guest init script
runs as PID 1 and execs a fixed supervisor command line, and libkrun's
krun_set_exec receives an empty argv, so there was no channel for driver-owned
supervisor arguments. The supervisor's proxy flags deliberately have no
environment fallback, and build_guest_environment merges user-supplied
environment, so the guest env is not a safe transport either.

Add a driver-authored argument file, mirroring the existing init.d manifest:
the driver writes /opt/openshell/supervisor-args into the overlay upperdir on
every launch and the guest reads it verbatim, one argument per line, appending
it to every supervisor exec. It is written even when empty, which is what makes
the channel unforgeable -- the upperdir always shadows the read-only image
layer, so an image can neither supply its own arguments nor disable the
operator's by omitting the file. Because both launch backends exec the same
init script, this covers libkrun and QEMU without touching either.

A microVM has no bind mounts or container secrets, so the credential and CA
bundle are staged into the per-sandbox overlay the way the gateway JWT already
is: credential root-only at 0600, CA at 0644, both rewritten every launch so a
removed setting clears prior material, and both deleted with the sandbox state
directory. This places the credential at rest in the overlay image on the
gateway host, which differs from the Podman secret model and is documented as
an explicit security consideration.

Validation is fail-closed and shared: a new
openshell_core::driver_utils::validate_upstream_proxy_settings holds the
pairing rules the Podman driver established, and both the gateway and the
driver call it so an invalid table names the offending key instead of
surfacing as an opaque driver-readiness timeout.

Guest egress leaves through gvproxy, so a proxy on the gateway host's loopback
is reachable only through host.openshell.internal; the guest to gateway
callback is unaffected.

Closes NVIDIA#3088

Signed-off-by: Philippe Martin <phmartin@redhat.com>
Two review findings on the corporate forward proxy support for microVM
sandboxes.

The driver read the operator's proxy_ca_bundle with an unbounded fs::read and
accepted it on a substring match for the PEM BEGIN CERTIFICATE marker. A
special file such as /dev/zero therefore grew driver memory without bound on
every authorized sandbox create, and a PEM block holding invalid DER passed
the host check but contributes no trust anchor in the guest, so every
supervisor would fail after boot with an error attributed to the sandbox
rather than to the setting.

Move the read into openshell-core as read_upstream_proxy_ca_bundle_file: it
reuses the credential reader's bounded-read path (non-regular files rejected
on fstat, size capped, read bounded even if the file grows), then requires at
least one anchor that RootCertStore::add_parsable_certificates accepts. The
supervisor's own reader now delegates to it, so host acceptance and guest
acceptance are the same function and cannot drift.

The published host-loopback recipe was written for libkrun only. gvproxy NATs
host.openshell.internal to the gateway host's 127.0.0.1, but GPU sandboxes run
on the QEMU/TAP backend where that name resolves to the TAP host address and
the driver's own nftables input chain accepts only the gateway port from the
guest — no proxy on the gateway host is reachable there at any bind address,
so an operator following the generic recipe lost all proxy-required egress
while configuration validation succeeded.

Scope the recipe to libkrun in every reference and reject a gateway-host proxy
URL when a launch plan resolves to QEMU, naming the reason, instead of booting
a sandbox whose policy-approved CONNECTs all time out.

Signed-off-by: Philippe Martin <phmartin@redhat.com>

@johntmyers johntmyers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

PR Review Status

Thanks @feloy. I checked your replies against current head 38e83fad785099c25a56901d73dd08a5957f26a2: the shared bounded regular-file reader plus rustls trust-anchor validation resolves GATOR-dc1218b5-01, and the backend-scoped documentation and QEMU preflight resolve the original GATOR-dc1218b5-02 obligation. The follow-up review found one new blocker in that preflight: it does not identify the actual per-sandbox TAP host address and rejects gvproxy's unrelated private address instead.

Action required: @feloy, please address GATOR-38e83fad-01 and push an update.

Blocking findings:

  • GATOR-38e83fad-01: classify the selected QEMU TAP host address without rejecting an unrelated routable proxy address.

Carried findings:

  • None; both prior findings are resolved.
Gator metadata
  • Validation: Project-valid implementation of linked issue #3088.
  • Docs: Updated for the libkrun/QEMU reachability split; implementation parity remains blocked by the new finding.
  • Checks: Current-head Branch Checks and Helm Lint are pending; test dispatch remains deferred while review feedback is open.
  • E2E: test:e2e is required for VM networking and credential flow and will be dispatched after blockers are resolved.
  • Head SHA: 38e83fad785099c25a56901d73dd08a5957f26a2
  • Base SHA: 74960ebfaeec4673885089ed995fad902459749f
  • Merge base SHA: 74960ebfaeec4673885089ed995fad902459749f
  • Patch ID: a2aceafceaca70aa799bc9a7503cadbae7203036
  • Gator payload: 8
  • Review mode: follow_up
  • Previous reviewed SHA: dc1218b51ac14f1315c1a633cae316ff45963549
  • Review budget exhausted: no
  • Maintainer decision required: no
  • Next state: gator:in-review

Comment thread crates/openshell-driver-vm/src/driver.rs Outdated
The gateway-host proxy guard added for the QEMU/TAP backend classified the
wrong set of addresses in both directions.

It ran at the top of configure_qemu_launch_plan, before the subnet allocation
that settles plan.host_ip, so it could not compare against the address the
guest actually reaches the host on. An operator pointing https_proxy at the
sandbox's own TAP host address, such as 10.0.128.1, passed the check, and the
driver's nftables input chain — which accepts only the gateway port from the
guest — then dropped every policy-approved CONNECT, which is exactly the
silent timeout the guard exists to prevent.

In the other direction it rejected 192.168.127.254 unconditionally. That
address is special only to libkrun/gvproxy; on QEMU/TAP it is an ordinary
address that may be routable through the guest's masqueraded egress, so the
guard refused a working configuration.

Run the check after the launch plan's network allocation, on both the
freshly-allocated and already-complete paths, and compare IP literals with
that sandbox's selected TAP host. Loopback literals, localhost, and the
documented host aliases that write_host_gateway_aliases seeds to the TAP host
still classify as the gateway host, and the failure names the address. The
gvproxy host-loopback constant returns to being a documentation anchor.

Signed-off-by: Philippe Martin <phmartin@redhat.com>
@feloy
feloy requested a review from johntmyers September 2, 2026 10:48
@johntmyers johntmyers added the test:e2e Requires end-to-end coverage label Sep 2, 2026
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown

Label test:e2e applied, but pull-request/3090 does not exist yet. A maintainer needs to comment /ok to test eb4f23630e44e1e8336412ea9fad3885bbb5ea3a to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

@johntmyers

Copy link
Copy Markdown
Collaborator

/ok to test eb4f236

@johntmyers johntmyers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

PR Review Status

Thanks @feloy. I checked your detailed response against current head eb4f23630e44e1e8336412ea9fad3885bbb5ea3a: the proxy preflight now runs after the QEMU TAP host address is finalized, compares parsed addresses with that selected host, no longer rejects gvproxy's unrelated 192.168.127.254 address, and covers both launch-plan directions. This resolves GATOR-38e83fad-01; the follow-up review found no new blockers.

Blocking findings:

  • No blocking findings remain

Carried findings:

  • None; GATOR-dc1218b5-01, GATOR-dc1218b5-02, and GATOR-38e83fad-01 are resolved.
Gator metadata
  • Validation: Project-valid implementation of linked issue #3088 with an explicit maintainer-requested phase.
  • Docs: Updated for the VM corporate-proxy UX and backend reachability contract.
  • Checks: Current-head Branch Checks, Helm Lint, and Branch E2E workflows are running.
  • E2E: test:e2e applied; the copy-PR mirror was authorized for this head and Branch E2E run 33622651241 is in progress.
  • Head SHA: eb4f23630e44e1e8336412ea9fad3885bbb5ea3a
  • Base SHA: 74960ebfaeec4673885089ed995fad902459749f
  • Merge base SHA: 74960ebfaeec4673885089ed995fad902459749f
  • Patch ID: fc66764b47b57c726c06c8e9029441978d2bae30
  • Gator payload: 8
  • Review mode: follow_up
  • Previous reviewed SHA: 38e83fad785099c25a56901d73dd08a5957f26a2
  • Review budget exhausted: no
  • Maintainer decision required: no
  • Next state: gator:watch-pipeline

@johntmyers johntmyers added gator:watch-pipeline Gator is monitoring PR CI/CD status gator:approval-needed Gator completed review; maintainer approval needed and removed gator:in-review Gator is reviewing or awaiting PR review feedback gator:watch-pipeline Gator is monitoring PR CI/CD status labels Sep 2, 2026
@johntmyers
johntmyers added this pull request to the merge queue Sep 2, 2026
Merged via the queue into NVIDIA:main with commit 857af42 Sep 2, 2026
82 of 84 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gator:approval-needed Gator completed review; maintainer approval needed test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(vm): support corporate HTTP forward proxy egress for microVM sandboxes

2 participants